Mean Model Clustering

In this paper, an effective and efficient model-based 2-clustering approach using Fisher kernel based similarities [2][3] is proposed. The Fisher kernel is derived from the knowledge of the family of the distribution generating the data. In the proposed approach, we refrain from trying to estimate the parameters of the unknown mixture distributions generating the data using the EM algorithm. Instead, the Fisher kernel values are computed based on the mean model obtained by assuming that the entire data has been generated from a single member of the family, rather than a mixture, and thereby systematically ignoring the so-called hidden random variables. The Fisher kernel similarities in terms of the mean model have an interesting property that turns out to be quite useful for data clustering. Consider a set ofN data points sampled independently following a mixture of two distributions with densities belonging to the parametric family f(x; ) parameterized by d real valued variables = [ 1; ; d℄ 2 , where is an open subset of Rd . The family of generative models f(x; ) define a d-dimensional Riemannian manifold S whose natural Riemannian metric is given by the Fisher Information matrix [1]. Let the two generating distributions be denoted by f(x; +) and f(x; ). Knowing fully well that the data has actually been sampled following a mixture of two distributions, we intentionally assume that the entire data has been sampled following a single distribution, i.e., the mean model f(x; 0), belonging to the same family. Note that the parameters 0 can be obtained in almost all practical cases directly from the given data by a simple maximum likelihood estimation. The concept of a distribution dependent Fisher kernel was introduced in [2]. Let K̂0( ; ) represent the Fisher kernel with respect to the estimated mean model. We show that if x1;x2 are samples drawn independently from f(x; +), and y1;y2 are samples drawn independently from f(x; ), then Pr[E[K̂0(x1;x2)℄ 0℄ = Pr[E[K̂0(y1;y2)℄ 0℄ = 1 where is a special coordinate system on the manifold S related to by a one-one differentiable map and ̂0 = (̂0), the expectation is over the joint distribution of the samples, and the probability is over the randomness of ̂0. Further, Pr[E[K 0(x1;y1)℄ < 0℄ > 1 tr(G 0G 1 ̂0 )=Nr2 0 , where G 0 and Ĝ0 are respectively the Fisher Information matrices at the points in the manifold corresponding to the true mean model and the estimated mean model and r0 is a positive constant. The lower bound on this probability goes to 1 as N !1. Probabilistic lower bounds on the actual kernel values K̂0(x1;x2) and K̂0(y1;y2) being positive, and K̂0(x1;y1) being negative can be computed for a given family and its value is primarily determined by the concentration of the sufficient statistic random variable corresponding to the parametric family under consideration around its expectation. Any clustering algorithm that can use the mean model Fisher kernel similarities to separate the data can be used for a mean model clustering scheme. We propose a simple graph-based mean model clustering scheme. Given the set of data-points, sampled from either f(x; +) or f(x; ) with uniform priors, let A denote the pairwise kernel similarity matrix of the points so that [A℄ij = K̂0(xi;xj). Considering A to be the adjacency matrix of a graph whose vertices are the data points, a 2-partitioning of the graph following a min-cut objective gives the graph-based clustering of the data.